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Abstract 

As a leader in space technology research and development, NASA is continuing in the development 
of the Technology Estimating process, initiated in 2012, for estimating the cost and schedule of low 
maturity technology research and development, where the Technology Readiness Level is less than TRL 
6. NASA’s Technology Roadmap areas consist of 14 technology areas. The focus of this continuing 
Technology Estimating effort included four Technology Areas (TA): TA3 Space Power and Energy 
Storage, TA4 Robotics, TA8 Instruments, and TA12 Materials, to confine the research to the most 
abundant data pool. This research report continues the development of technology estimating efforts 
completed during 2013-2014, and addresses the refinement of parameters selected and recommended for 
use in the estimating process, where the parameters developed are applicable to Cost Estimating 
Relationships (CERs) used in the parametric cost estimating analysis. This research addresses the 
architecture for administration of the Technology Cost and Scheduling Estimating tool, the parameters 
suggested for computer software adjunct to any technology area, and the identification of gaps in the 
Technology Estimating process. 



Introduction and Background 

This research covers the period from July 2013 to February 2014 and was a continuation of the 
work to implement a capability for providing a technology estimating process, initiated in 2012, and 
sponsored by the NASA Office of Evaluation, Cost Analysis Division (CAD) in collaboration with the 
Space Technology Mission Directorate, and Game Changing Development Program Office. 

The research goal was to improve NASA’s ability to estimate the cost of technology research and 
development for low maturity technologies. This technology estimating research evolved by selection of 
critical parameters that were expected to influence the cost of a technology research and development, a 
continuation of collecting actual data for completed technology efforts, and then conducting mathematical 
and statistical analyses to generate cost estimating relationships between the parameters and use of these 
data in an analysis to determine the cost and schedule of technology development for Technology 
Readiness Level less than 6. (1) 

Recognizing that NASA currently has 14 Technology Roadmap Areas, the effort for constructing 
a database of completed projects for the full 14 Technology Areas was beyond the scope of the research. 
Strategically, only a few select technologies were to be investigated and these included Technology Areas 
(TAs): Space Power (TA03) and Energy Storage, Robotics (TA04), Instruments (TA08), and Materials 
(TA12), as these provided the highest number of potential projects for use in evaluating the parameters 
considered, and were determined to be good candidates for fulfillment of data acquisition. 

A diagram, Figure 1, was prepared to map the continuation of the procedural process followed 
during the research project and includes a series of activities, parallel analysis, workshops, an interim 
review, and internal reviews within the NASA Cost Community to continue with the identification of 
issues and the development of the Technology Estimating Research project. This structure facilitated the 
necessary NASA Cost community input and acceptance of the process, and thereby allowing a high 
profile platform for the internal review. More specifically, the process was adapted to include dual 
processes in the Technology Cost and Schedule Estimating (TCASE) tool. 



Figure 1. Technology Estimating Process Diagram, 2013-2014 
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Data is critical for the Technology Estimating process. Obtaining this data in the form of cost 
estimating parameters for each of NASAs R&D projects was considered fundamental to the Technology 
Estimating process and continued development of the Technology Cost and Schedule Estimating 
(TCASE) tool. NASA’s TechPort was identified in this research as the most reliable resource and 
repository for this technology estimating data. TechPort is a portal developed by NASA, as a software 
system to capture, track, and manage NASA's portfolio of technology investments. 

Evaluation of Technology Estimating Parameters 

The Technology Estimating research conducted during 2012-2013 determined that the best 
method suited for use as an estimating method for estimating technology research and development 
(R&D) was the parametric analysis. The research then focused on defining those parameters, which best 
represent the drivers for cost and schedule in technology R&D and ultimately incoiporating those 
parameters in the parametric analysis as Cost Estimating Relationships (CERs). 

Parametric estimating is a statistical relationship method developed between historical costs and 
program, physical, and performance characteristics. The parametric method relates cost to one or more 
technical, performance, cost, or program parameters, using a statistical relationship. The goal of 
parametric estimating is to create a statistically valid cost estimating relationship using historical data. <2) 
The foundation on which parametric estimating is based is the use of CERs by which the analysis is 
performed. 

An analysis was conducted for approximately 600 project records in only four of the 14 
Technology Areas (i.e., TA03, TA04, TA08, and TA12) as a test bed, and to reduce the set of technology 
project data parameters to those key parameters determined to be most informative in terms of predicting 
target variables for both total project cost and total project duration. This reduction in the parameter set 
number makes the data collection more practical. In addition, the reduced number of parameters provides 
for increased efficiency and affords a practical step in the process for creating a predictive model for cost 
& duration. 

The analysis of the historical technology project data in the TCASE database was also performed 
to help determine which parameters were believed to be most useful for cost and duration prediction. As a 
predictor, the parameter must be a schedule and cost driver. As a driver, a parameter that tends to increase 
or decrease schedule/cost, or has been identified to be highly correlated with schedule/cost. To qualify as 
a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense. 

The methodology used in the analysis included an examination of each technology project data 
parameters on a one-at-a-time basis. First, each parameter was taken and then the set of 600 plus data 
records were subdivided by all levels of that parameter. Next, a statistical test was performed to determine 
whether there was a significant difference, i.e. beyond simple randomness in the data, between the 
distributions of cost and/or duration for projects in these subdivided groups. The tendency of a parameter 
to subdivide the full dataset into groups with significantly different costs and/or durations was evaluated 
to indicate if a parameter was a good choice for use in predictive modeling. The statistical testing 
described here was just one of several factors that may ultimately determine which technology project 
data parameters were to remain in the final set. 

The statistical analysis used for the examination of the parameters was the Kruskal- Wallis one- 
way analysis of variance. This statistical method of analysis is a non-parametric test that is applied much 
like the Analysis of Variance (ANOVA) test. 1 ” Non-parametric means that this test does not assume the 
data to be tested conform to any particular distribution shape or characteristics, e.g. does not require input 
data to adhere to a normal distribution. The ANOVA test is best suited for test data that has been found to 
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be approximately normally distributed ( although the test is robust to violations of this assumption as long 
as the sample size is large). The TCASE data tested was not always normally distributed. Kruskal- Wallis 
has no such assumption of normally distributed test data and is non-parametric. 

Where the Kruskal- Wallis test did not apply, the Two Sample t-Test was used for the parameter 
analysis. In cases where the parameter were dichotomous, e.g. Yes/No, Push/Pull, etc., the t-test was 
determined to be an appropriate method and will return identical or nearly identical results when 
compared with the more involved tests such as ANOVA. The analysis was performed to reveal those 
parameters when considered individually, partition the full sample set into groups with statistically 
significant distributions of project cost and/or duration. In statistical terms, the null hypothesis is a 
condition where the distributions are the same across all levels. The alternative hypothesis is that the 
distributions are not the same. 

In summary, the statistical test results presented in Table 1, although interesting, were not 
predisposed to fulfill the notion that a useful regression model could be identified or a parametric model 
could be created using these data. The analysis indicates that the parameters identified as “rejected null 
hypothesis” could potentially contribute to a future parametric model that is statistically valid and well- 
conceived. 


Table 1 

Parameter Data Analysis Results 

Parameter 

Name 

Parameter 

Type 

Sample Size 
(cost; 
duration) 

Test Type 
(p-value = 
0.05) 

Test Outcome 
Target = 
COST 

Test Outcome 
Target = 
DURATION 
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Technology Area 

Categorical (4 levels) 

n = 620; 583 

Kruskal- Wallis 

✓ 


✓ 


Technology Readiness Level 

Categorical (27 levels) 

n = 541; 535 

Kruskal-Wallis 

✓ 


</ 


System Hierarchy Level 

Categorical (5 levels) 

n = 609; 573 

Kruskal- Wallis 

✓ 


✓ 


Lead NASA Center 

Categorical (8 levels) 

n= 166; 139 

Kruskal-Wallis 

✓ 



«/ (a) 

RD 3 

Categorical (5 levels) 

n= 101; 99 

Kruskal-Wallis 

✓ 


✓ 


Hardware vs. Software 

Categorical (5 levels) 

n = 55; 54 

Kruskal-Wallis 


✓ 


✓ 

Team Experience 

Categorical (4 levels) 

n = 53; 52 

Kruskal-Wallis 


</ 


✓ 

2 

AD 

Categorical (9 levels) 

n = 53; 52 

Kruskal-Wallis 




✓ 

Unplanned Events 

Dichotomous 

n = 44; 43 

Two Sample t-Test 


✓ 


✓ 

Technology Push vs. Pull 

Dichotomous 

n = 63; 62 

Two Sample t-Test 


</ 


✓ 

Evolutionary vs. Revolutionary 

Dichotomous 

n = 52; 51 

Two Sample t-Test 


* ( a ) 


✓ 

Facilities and Infrastructure 

Dichotomous 

n = 49; 49 


N/A (could not test) 



Budget Constraints/Disruptions 

Dichotomous 

n = 46; 55 

Two Sample t-Test 


(a) 


✓ 


Note: N/A = Not applicable, (a) = Small relaxation in p-value (significance level) would have led to rejection of the null 
hypothesis. 


The results of the analysis conducted using the Kruskal- Wallis and t-Test contributed to the 
elimination or reconfiguration of the following parameters: push versus pull, evolutionary versus 
revolutionary; single project versus portfolio; budget constraints/disruptions, Advancement Degree of 
Difficulty (AD 2 ), and hardware versus software. The parameter for hardware was eliminated and software 
was expanded to include additional parameters based on information to be acquired by submission of 
Software Management Plans in accordance with NPR 7150.2A. 
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A qualitative analysis was performed to establish the preferred parameters to be used in 
Technology Estimating. The qualitative analysis is included in Appendix A. 

Based on the statistical analysis conducted, Table 1 and the qualitative analysis, Appendix A, the 
cost estimating parameters recommended for use in estimating the cost and schedule of technology R&D 
with low maturity, i.e. TRL less than 6, are included in Table 2. Each parameter was classified as a 
qualitative or quantitative data type. Seven of the 15 Technology Estimating parameters are included in 
TechPort. 


Table 2 


Technology Estimating Parameters 


Category 

Parameter 


Data Type 

Included in TechPort 



Qualitative 

Quantitative 


Project Characterization 

Technology Area 

✓ 


✓ 


System Hierarchy 

✓ 




Defining system characteristics 

✓ 

✓ 



Computer Software 





Project Type 

✓ 




Software Class* 

✓ 




Primary Language 

✓ 




Number of Requirements 


✓ 



Software Size 


✓ 


Projects Results 

Key Performance Parameter(s) 


</ 

<✓ 


Level of Effort 


✓ 



Actual Cost 


✓ 

✓ 


Schedule (Start and End Date) 



✓ 

Development Metrics 

TRL 

✓ 


✓ 


RD 3 

<•/ 



Project Execution 

Funding source(s) 

✓ 


✓ 


Perfonning Organizations 

✓ 


✓ 


Team Experience 

s/ 




Facilities and Infrastructure 


✓ 


Programmatic Factors 

Unplanned Event(s) 

✓ 

✓ 



Note: TRL = Technology Readiness Level; RD 3 = Research and Development Degree of Difficulty; 
* = Class A, B, C, D, or E in accordance with NPR 7150. 2A 


Cost Estimating Relationships 

The TCASE cost and schedule tool is currently an analogy based cost and schedule estimating 
tool. Its future development includes the application development and use of parametric analysis once 
sufficient project data is available. The continued increase of data will allow the analysis to be conducted 
for subsequent establishment of Cost Estimating Relationships (CERs). Obtaining an adequate number of 
data in each of the Technology Areas and sub-TAs present has presented a challenge for derivation of 
CERs that span the four Technology Areas that were initially investigated, i.e. TA03 Robotics, TA04 
Power, TA08 Instrumentation, and TA12 Materials. It was determined that CER development for the 
balance of the remaining Technology Areas will be accomplished after cost and schedule data can be fully 
acquired from sources such as TechPort. 

The validity of a CER will usually be judged by its regression statistics, which measure the 
accuracy of the fit of the CER to the sample data points used in developing the CER. The most commonly 
used regression statistic is the coefficient of determination (R 2 ), sometimes described as “goodness-of- 
fit”. A CER that perfectly predicts each sample point in the database would have an R 2 of 1.0. For CERs, 
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an R 2 value of 0.9 or better is considered desirable, although in practice, CERs with R 2 values of 0.8 or 
better may be accepted for use in a cost estimate/ 4 ^ 

The format of the CERs to be developed for Technology Estimating should follow The Project 
Cost Estimating Capability (PCEC). In 2013, NASA began investigating the development of changes to 
the existing cost estimating data platforms. The Project Cost Estimating Capability (PCEC) was identified 
as a new framework that will replace NASA Air Force Cost Model (NAFCOM) as the standard NASA 
capability for estimating the cost of new spaceflight hardware systems during concept exploration and 
refinement. When the development is complete, the PCEC will be an Excel™ based architecture that 
combines a user interface running Visual Basic Algorithm (VBA) with Work Breakdown Structure 
(WBS) and CER libraries. 

Technology Estimating Process 

The Technology estimating process includes the acquisition of data for use the Technology 
Estimating Cost and Schedule Estimating tool. Multiple sources of data have been investigated and the 
most robust source of data determined viable for low maturity technologies, i.e. TRL less than 6. 

TechPort is NASA’s premier information portal for research and development technology. It has 
been identified as a significant data source for Technology Estimating. Of the 15 technology estimating 
parameters identified for support of Technology Estimating (Table 3), seven parameters are already 
included in TechPort. Eight additional parameters have been identified that support the Technology 
Estimating process should be included in TechPort. To facilitate the addition of eight additional 
Technology Estimating parameters in future changes to TechPort, Table 3 indicates a suggested order 
sequence for their inclusion. A sequencing of these additional parameters into TechPort would also help 
to integrate changes systematically, adjust users to these new parameters methodically, and help reduce or 
minimize potential errors in their application. 

The priority for order of sequence was a ranking from “1” the highest priority to be implemented 
first, to “3” the lowest priority to be implemented last. The rank and priority was based on: 

• The metric associated with the parameter, 

• The parameters’ perceived ease of application as observed during the Technology Estimating 
research project, 

• Experience for the parameters’ use and application by Technology Managers as observed during 
the research, and 

• The parameters’ level of definition. 
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Table 3 

Technology Estimating Parameters to Be Added in TechPort 

Category 

Parameter 

Data Type 


Priority and 
Sequence for 
Inclusion in TechPort 



Qualitative 

Quantitative 


Projects Results 

Level of Effort 


✓ 

1 

Development Metrics 

RD 3 



1 

Project Execution 

Team Experience 

✓ 


1 

Project Characterization 

System Hierarchy 

✓ 


1 

Project Characterization 

Defining system characteristics 

✓ 

✓ 

2 

Project Characterization 

Computer Software 





Project Type 



2 


Software Class* 

✓ 


2 


Primary Language 

✓ 


2 


Number of Requirements 


✓ 

2 


Software Size 


✓ 

2 

Project Execution 

Facilities and Infrastructure 


✓ 

3 

Programmatic Factors 

Unplanned Event(s) 

✓ 

✓ 

3 


Note: TRL = Technology Readiness Level; RD 3 = Research and Development Degree of Difficulty. * = Class A, B, C, D, or E in accordance 

with NPR 7 1 50. 2A 


The TCASE tool should include projects from TechPort for use in Technology Estimating. Once 
cost and schedule data are obtained from TechPort, for all of the NASA Technology Roadmap areas 1 
through 14, and a level of confidence is obtained for importing data and Technology Estimating, then 
additional Technology Estimating parameters, as identified in Table 3 should be incoiporated sequentially 
according to its indicated priority. 

Administration of Technology Estimating Process 

The process for technology estimating includes the use of data for completed research and 
development projects with low maturity. A chart, Figure 2, indicating proposed administration activities 
was prepared to reveal the path of project data from initiation of the project by the Technology Owner to 
the project documentation data sources such as TechPort, Cost Analysis Data Requirements (CADRe) by 
NPR 7120. 5E, and Software Management Plans submitted in accordance with NPR 7150.2A. Other 
project data have been migrated to TCASE such as those data sources identified in Table 4. 
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Table 4 




Data Sources Used in this Research <5) 



Short Name 

Description 

Year 

Entries 

ETDP 

NASA Exploration Technology Development Program 

2012 

42 

ESTO 

NASA Earth Science Technology Office 

2012 

138 

SBIR 

Current NASA SBIR Technologies 

2012 

51 

GCD 

Game Changing Development Program Office Technology Projects 

2012 

35 

ESMD 

NASA Exploration Systems Mission Directorate ESAS Technologies 

2005 

304 

Tech Tool Box 

ATLAS (Advanced Technology Lifecycle Analysis System) 

2002 

6 

Ext Tech Data 

Tauri Group research into External Government Technologies (EGT within 
the MATCH relational database) 

2012 

28 

Cx ETDP 

Constellation Exploration Technology Development Program 

2007 

19 

NTID 

NASA Tech Inventory Database 

2004 

991 

Hist SBIR STTR 

Historical SBIR and STTR data 

2012 

1191 

RLV Tech DB 

NASA RLV Technology Database 

1993 

64 

HEOMD TechDev 
Team 

Estimates of HEOMD needed technology developments 

2012 

78 

ETDP from 
MATCH 

Mapping Applicable Technology To Exploration Challenges 

2007 

21 

External from 
MATCH 

Mapping Applicable Technology To Exploration Challenges 

2007 

192 

JSS TBCC 

NASA/Air Force Joint Systems Study TBCC Technologies 

2010 

18 


Individual project data from the data sources are then migrated to One NASA Cost Engineering 
(ONCE) for use in the TCASE tool. The data for TCASE can be reviewed by an Administrator who will 
review data quality, assess parameters and perform any needed data scrubbing; tool updates, distribution 
and version control; and CER development. Any data that requires retirement due to age or applicability 
will be evaluated by the TCASE Administrator and notify the Technology Owner of any errors, omissions 
or issues as may be determined helpful. 



Technology Estimating 





Figure 2. Technology Estimating process 


Data Migration from TechPort to the TCASE Tool 

TechPort is NASA’s repository of information on each individual project within NASA R&D 
programs. The TechPort R&D projects are searched by several parameters including seven of those 
identified for Technology Estimating. Representative parameters included are the project name, start and 
end date, NASA technology roadmap area, and key performance parameter, etc. After the TechPort user 
completes a selection process, the multiple attributes for each of the selected projects are then 
downloaded to an EXCEL™ spreadsheet. 

Currently, due to gaps observed in the project records for a project in TechPort, the completeness 
of Technology Estimating parameters may not always be fulfilled or included for the respective project 
data file and therefore a parameter field may be bla nk or omitted. A February 2014 survey of project 
records for “completed” projects indicated that a critical parameter such as “cost” data may not have been 
imported to TechPort by the Technology Owner. The lack of cost data in TechPort impedes the 
Technology Estimating capability. Currently, the TCASE (version 5.0) tool was designed to be 
configured to accept a download of the projects EXCEL™ attribute file directly from TechPort. 

Future anticipated refinement of ONCE will be expected to be configured to allow the project 
data exported or downloaded directly from TechPort to ONCE. As the data will be “mirrored” from 
TechPort to ONCE, no modifications to TechPort will be necessary with respect to Technology 
Estimating. As ONCE will be the data repository for Technology Estimating, the project data will be 
normalized and evaluated for use in Technology Estimating. If errors are found in the data, which are 
located in ONCE, questions arise, or the data needs to be retired, then the TCASE Administrator will 
make the changes in ONCE or notify the respective Technology Owner directly as may be required. 
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Data Migration from CAD Re to the TCASE Tool 

CADRe is NASA’s primary technology data file for projects meeting requirements of NPR 
7120. 5E. The CADRe is prepared by the CAD using existing project data prepared during the life-cycle 
review process. By capturing key information, the CADRe tracks and explains changes that occurred 
from one milestone to the next, and it helps the project manager record in an Agency document all the 
events that occurred during the project both internal and external. The CADRe is a project owned 
document and is signed by the project manager; therefore, it does not include any independent 
assessments or evaluations, or opinions about the project. It simply records the known configuration at the 
specific milestone. The CAD provides the necessary funding and contractor support to prepare the 
document on behalf of the project using existing project documentation prepared during the milestone 
review process. Composed of three parts, the CADRe captures detailed programmatic, technical, and cost 
data in a standardized format. The document is prepared six times during the life cycle of a project at 
major milestones (SRR, PDR, CDR, SIR, Launch, End of Mission). Currently, CADRe and the data being 
included in the database are potentially migrating to the PCEC, and ONCE. Future refinement of ONCE 
should allow the data for technologies (projects) with TRL less than 6 to be included in TCASE. 

Data Migration from NPR 7150.2A Software Management Plans to the TCASE Tool 

Observations made during the research indicated that a significant portion of the technology 
effort might be attributed to the development of computer software. Software applications may be 
required for support other Technologies, or required as an integral part of the technology R&D. 
Segregating this software effort could be accomplished via use of the Software Management Plans 
(SMPs) required by NPR 7150.2A. The SMPs contain pertinent data about the software and includes the 
parameters that drive the cost and schedule. As identified in Table 3, the Technology Estimating 
parameters identified for computer software include project type, software class, primary language, 
number of requirements, and software size. 

Data Tracking 

Technology R&D projects may be initiated within any number of potentially 42 different NASA 
R&D programs. Technology owners may initiate a project in one NASA Technology Roadmap area and 
the project may take on follow on work that could progress into later phases of work, “morph” into other 
technology areas, or there may be successive parent-daughter technology projects. Tracking the initial 
R&D technology projects and its follow-on work is paramount to obtaining the total cost and schedule 
of a particular technology as would be accomplished in Technology Estimating. 

Based on the findings from this research, tracking a technology projects cost, schedule, and other 
pertinent project data through the succession of phases has been identified to be complicated by a number 
of items. These include: 

• The Principal Investigator left the project where there is no information related to their 
succession, 

• The R&D project work did not result in a formal technical paper, 

• Contract or research grant identifying numbers were not identified in the research technical 
paper, 

• The project administration of the research did capture the R&D project administration 
information in a centralized NASA location similar to TechPort, 

• The project records were archived and are difficult to retrieve, 

• Funding made by two separate NASA organizations on the same research project could not be 
connected without identification of the original contract number, or WBS numbers were 
unavailable, 

• Project funding was postponed. 
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• The research scope changed, or was transferred to another program, or 

• Information in the acknowledgment section of the technical paper did not indicate sufficient 
information to connect with previous or succession work. 

Tracking technology is best accomplished from the beginning of the R&D work where the 
Technology Owner’s researcher is notified of the requirement to complete and submit all the 
applicable information to TechPort at the conclusion of the work. This new requirement would be 
identified in two primary documents. The first is NASA’s Guidebook for Proposers Responding to a 
NASA Research Announcement (NRA) or Cooperative Agreement Notice (CAN), January 2013 Edition, 
and the second is NPR 5800.1 Grant and Cooperative Agreement Handbook, Section A, Part 1260, Grants 
ad Cooperative Agreements, § 1260.77, Closeout procedures, located at 
https://prod.nais.nasa.gov/pub/pub library/grcover.htm. 

During this research, it was identified that some technical papers reviewed did not always reveal a 
clear line of technology succession even though multiple papers were identified by author and successive 
dates. A possible solution to help reveal technology and their successive technologies is a new 
requirement to be incorporated in NASA’s Science and Technical Information (STI) paper submission 
process. The NASA documents to be modified to help in the tracking effort would include NPD 2200. IB, 
Subject: Management of NASA Scientific and Technical Information, Effective Date: November 19, 

2009, Expiration Date: November 19, 2014; NPR 2200.2C, Requirements for Documentation, Approval, 
and Dissemination of NASA Scientific and Technical Information, Effective Date: April 19, 2011, 
Expiration Date: April 19, 2016. Additional research should be conducted to determine the best method 
for tracking technologies and identified in the “Acknowledgement” section of the technical paper. 

Tracking technology R&D by the use of Work Breakdown Structure (WBS) as a means for 
classification of technology was not considered in this research. It was found that the majority of 
NASA‘s individual R&D low maturity projects with TRL less than 6 typically did not appear to have a 
WBS. Portfolios of low maturity R&D technology projects were thought to follow the portfolio 
requirements of NPR 7120.8, Chapter 5 by use of a Portfolio Project Plan as required by NPR 7120.8, 
NASA Research and Technology Program and Project Management Requirements (w/change 3 dated 04/18/13) 
Section 5.2.5 Project Implementation, 5. 2. 5.1. In the Implementation phase, the R&T Portfolio project lead executes 
the R&T Portfolio Project Plan, which usually consists of one or more portfolio cycles (see section 5. 2. 5. 7). 

These Portfolio Project Plans were not reviewed in this Technology Estimating research as the effort 
was outside the research scope. Further research should be conducted to determine the extent of 
tracking requirements as could be identified in the administration of NASA ’s low maturity R&D 
projects by NPR 7120.8. This tracking of individual projects, as managed in the Portfolio Project Plan 
could prove useful and whether WBS for individual projects over portfolios would prove valuable to 
Technology Estimating. 

For reference, the WBS Template worksheets in the PCEC Library vO have been formatted with a 
blue tab colors and each WBS worksheet contains a WBS. The following three WBS types are included 
in the PCEC: 1) NASA NPR 7120.5E WBS which is the NASA WBS. At this time the NAFCOM/PCEC 
CERs do not estimate to the WBS structure and is included in the PCEC for reference; 2) the CAD 
Requirements (CADRe) WBS which is used to collect data for NASA CADRe documents and is included 
in the PCEC for reference.; and 3) NFC12 WBS where it was derived from NAFCOM. The current 
PCEC/NAFCOM CERs align to these WBSs. Each template represents one of several major categories of 
systems produced by NASA. 
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Conclusion 

Technology Estimating and the TCASE tool will provide a significant capability to Technology 
Managers and Technology Owners. The use of TCASE as an analogy based estimating tool has been 
demonstrated and the tool is continuing to be improved. Once additional project data has been included 
from sources such as NASA’s TechPort, the value of the tool as an aid in estimating the cost and schedule 
of NASA R&D technology will increase greatly and its popularity for use in the cost community should 
expand. 

A critical gap in the Technology Estimating process was identified. The significant gap includes 
tracking initial and successive R&D technology projects. Closing this gap for tracking an initial project 
and a project’s follow-on work was determined to be paramount for obtaining the total cost and schedule 
of a particular technology. It was found during this work that additional research should be conducted in 
three primary areas to help support R&D technology project tracking, thus directly supporting the 
Technology Estimating process: 

• The first includes study to determine the best method for tracking technologies in technical 
papers. For example, these may include new requirements for specific notations to be identified in 
the “Acknowledgement” section of a technical paper. Further, where there is noted to be a 
succession of papers under the same contract, an annotation could be strategically identified. The 
implementation of these tracking requirements could be applied in NPD 2200. IB and NPR 
2200.2C, 

• The second area of additional study would be to determine the best methodology for tracking 
technology as it pertains to including project data in TechPort. It is suggested that tracking 
technology may best accomplished from the beginning of the R&D work, where the Technology 
Owner’s researcher is notified of the requirement to complete and submit all the applicable 
information to TechPort at the conclusion of the work, i.e. during closeout. Consideration of the 
appropriate changes to NPR 5800.1 could improve tracking of technology projects and support 
Technology estimating as Technology Owners could be required at the beginning of their work 
to complete data input to TechPort at the conclusion of the project. 

• The third area of additional study to be conducted would be to determine what technology 
tracking requirements that could be applied to the Portfolio Project Plan as required by NPR 
7120.8, and if the NASA WBS can be successfully applied to low maturity technology projects. 

The technology estimating parameters used in the estimating process were analyzed and based on 
both a statistical and a qualitative analysis. A total of 15 final Technology Estimating parameters have 
been recommended. Seven of these 15 parameters were identified to be already included in TechPort. A 
sequence and priority for the inclusion of the eight additional parameters into TechPort was identified. 

The eight additional parameters include System Hierarchy, Defining system characteristics, Computer 
Software, Level of Effort Research and Development Degree of Difficulty, Team Experience, Facilities 
and Infrastructure, and Unplanned Event(s). The addition of these parameters into TechPort should 
progress as a phased sequence where the benefit for including these eight additional parameters into 
TechPort will significantly outweigh the cost for their implementation. 
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Appendix A 

Qualitative Analysis of Technology Estimating Parameters 


14 



The following is a list of the parameters recommended for use in Technology Estimating process. The 
attribute of the parameter was identified as either qualitative or quantifiable and the use of both type 
would help describe the technology. Further, a ra nk for ease of administration of the parameter was 
identified. This ease of administration was based on observations made by the Research Team throughout 
the project. Generally, the guiding tenets for development of Technology Estimating parameters include 
that an item is a schedule/cost driver if a parameter tends to increase or decrease schedule/cost or is highly 
correlated with schedule/cost. 

Following each parameter is a narrative discussion. The discussion was provided to indicate any 
observations made during the research and help address the parameters suitability for use. 

1 . Parameter related to Project Characterization: Technology Area (TA) 


a. 

Metric Attribute: 

Qualitative 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. Technology areas in secondary and tertiary breakdown were sometimes difficult to apply by 
Technology Managers in all cases due to level of familiarity in application of the parameter. 

There is substantial documentation used to define these TAs and overtime, the use of the 
parameter will become easier for application and use. 

f. This parameter only applied to recent projects as the 14 TA Roadmap was promulgated within the 
last two years. 

g. Technology Area (TA) significantly helps to refine the Technology, however branching TAs can 
or may be questioned with regard to the accuracy of their application versus definition. 

h. Different individual TAs are naturally expected to have generally different schedule and costs. 

i. This parameter is included in TechPort. 

2. Parameter related to Project Characterization: System Hierarchy 


a. 

Metric Attribute: 

Qualitative 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. This parameter was relatively easy to be applied by Technology Managers. The distinction 
between the various level of system hierarchy can be close, depending on the system. 

EXAMPLE: If a satellite (mission i.e. “platform” or “system of systems”) had multiple sub- 
systems comprising the total science package, it could not readily be determined that a 
technology was itself a complete system or was it to be included in the larger science system or 
platform. 

f. Will benefit from a standard reference and definition to help clarify appropriate level in the 
hierarchy. 

3. Parameter related to Project Characterization: Defining System Characteristics 

a. Metric Attribute: Qualitative and Quantifiable 

b. Ra nk Ease of Administration: High 


15 



c. Schedule Driver: 

d. Cost Driver: 


Yes 

Yes 


DISCUSSION 

e. Provides the detailed characterization of TAs. 

f. Helps to characterize between technologies in same TA and for branching technologies from a 
single TA. 

g. This parameter includes metrics such as mass, volume, power, etc. 

h. With future data, the characterization could reveal that for some TAs i.e. 08 Instrumentation, that 
they follow Moore’s Law where “number of transistors on integrated circuits doubles 
approximately every two years” or possess smaller volumes, and less mass due to increased 
computing capability. 

4. Parameter related to Project Characterization: Software 


a. 

Metric Attribute: 

Qualitative and Quantifiable 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. Recommended that “Hardware” be dropped from original parameter (hardware versus software) 
as it did not provide significant correlation to amount or extent of software in all technologies. 

f. Suggested typical software cost drivers include: Project type, Software class, (i.e. A, B, C, D or 
E), Primary language, Number of requirements, and software size. 

g. Software development and its cost and schedule increases can be expected to be significant in the 
range of TRL 5 to 6. 

h. Technology papers for technologies with TRL < 6 indicate sometimes one or two persons 
exclusively dedicated to software when required. Extent of software used in R&D technologies 
was observed to be major influence to the cost and success of the technology. 

i. Data for software cost drivers are being collected as part of NPR 7150.2A Software Management 
Plan and can be used in Technology Estimating. 

5. Parameter related to Project Characterization: Key Performance Parameter (KPP) 


a. 

Metric Attribute: 

Qualitative and Quantifiable 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. This parameter normally includes any and all technology project objectives. 

f. The KPPs provide significant and relevant data about the project and its intended functional 
objectives. 

g. This parameter provides the description in sufficient detail to compare it against similar 
technology projects. 

h. This parameter also provides technical details such that branching applications may be 
understood. 

i. When adequate data is provided, technologies with similar objectives may be compared by their 
respective schedule and cost. 

j. The parameter is included in TechPort. 
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6. Parameter related to Project Results: Level of Effort 


a. 

Metric Attribute: 

Quantifiable 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

No 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. This parameter includes manpower estimate for NASA civil servant but does not capture the cost 
of universities, faculty, students, adjunct research investigators, etc. 

f. Currently, at the resolution of the project data obtained to date, the analysis indicates that the data 
cannot support that Level of Effort drives the schedule. 

g. Includes the capture the contracting costs in WYEs. 

7. Parameter related to Project Results: Cost 


a. 

Metric Attribute: 

Quantifiable 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. This parameter includes the total cost (actual). 

f. It is suggested that the cost per year be obtained and identified with the respective year. 

g. The cost that should be identified as the cost in the management information system. 

h. The parameter is included in TechPort. 

8. Parameter related to Project Results: Schedule 


a. 

Metric Attribute: 

Quantifiable 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. This parameter includes the total duration (actual). 

f. It is suggested that the total duration be identified by beginning and end dates. 

g. The duration that should be identified is the time in the management information system. 

h. The parameter is included in TechPort. 

9. Parameter related to Development Metrics: Technology Readiness Level 

a. Metric Attribute: Qualitative 

b. Rank Ease of Administration: High 

c. Schedule Driver: Y es 

d. Cost Driver: Y es 

DISCUSSION 

e. This parameter includes the Technology Readiness Level as it is submitted by the Technology 
Manager, PI, PM or Technical Paper. 

f. It is suggested that the TRL for a technology be assessed by an unbiased referee. 

g. It has been reported that the TRL may undergo changes in definition in the future. 

h. The parameter is included in TechPort. 
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10. Parameter related to Development Metrics: Research and Development Degree of Difficulty (R&D 3 ) 


a. 

Metric Attribute: 

Qualitative 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. This parameter includes five categories and can be difficult to administer. 

f. The use of this parameter in this research was impeded somewhat by subjectivity for its 
application. 

g. The statistical analysis reveal that the parameter may be a cost and schedule driver. 

1 1 . Parameter related to Project Execution: Funding Sources 


a. 

Metric Attribute: 

Qualitative 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. Comments from Technology Managers revealed that certain funding sources such IRAD 
programs do not provide enough time in the funding program to fully accomplish the technology 
project efforts. 

f. NASA undergoes a mix of funding programs, i.e. ETDP to GCPDO, etc. and the funding sources 
help to identify tracking of the technology and for branching of the technology. 

g. A gap exists for Tracking technologies from lower TRL to higher TRL, or from incubation to 
mission level. There is no way to identify if the work has been ceased due to funding, lack of 
capability or other reasons. The parameter was understood to be an imperative as it pertains to 
understanding technology infusion. 

h. The parameter is included in TechPort. 

12. Parameter related to Project Execution: Performing Organizations 


a. 

Metric Attribute: 

Qualitative 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

No 

d. 

Cost Driver: 

No 


DISCUSSION 

e. Identification of organizations involved helps to identify tracking of the technology and for 
branching of the technology. 

f. The identification of organizations and points of contact is an imperative for tracking technology. 

g. There was a perception that the parameter could be a schedule or driver if a certain organization 
is identified with the Technology. 

h. The parameter is included in TechPort. 

13. Parameter related to Project Execution: Team Experience 


a. 

Metric Attribute: 

Quantifiable 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 
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DISCUSSION 

e. Identification of the team structure goes directly to the schedule and cost driver of a technology. 

f. The team structure was not well documented in any of the data sources. 

g. The team makeup would be noted in project administration records and should be identified. 
Categories in this parameter include: Little or no experience conducting research, Experience 
conducting research but little or no experience with specific technology, Experience with research 
of this specific technology. Experience with this specific technology and innovative team. 

14. Parameter related to Project Execution: Facilities and Infrastructure 


a. 

Metric Attribute: 

Quantifiable 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. Identification of the facilities or infrastructure goes directly to the schedule and cost driver of a 
technology. 

f. The use of aircraft for testing was expected to be a large expenditure, however the cost was not 
normally documented. 

g. Some laboratories were noted in the literature to be specialized and there should have been a cost 
identified for these labs. 

h. Certain University test facilities by were noted in discussion with Technology Managers to 
severely impact the delivery date of a technology project. These types of changed conditions 
should be noted in project administration records and their impacts noted for use in Technology 
Estimating. 

15. Parameter related to Programmatic Factors: Unplanned Events 


a. 

Metric Attribute: 

Quantifiable 

b. 

Rank Ease of Administration: 

High 

c. 

Schedule Driver: 

Yes 

d. 

Cost Driver: 

Yes 


DISCUSSION 

e. This parameter includes budget constraints and disruptions including any and all significant 
events to the capture of any or all impacts to the project such as change orders, requirements 
change, or unforeseen conditions. 

f. This parameter should include any singular event(s) that that contribute to or impact the cost or 
schedule of the project. 

g. This parameter also includes any budget constraints and disruptions during the term of the work. 
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